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Abstract 

The  purpose  of  this  study  was  to  develop  Budget  At 
Completion  (BAG)  adjustment  factors  (BAFs)  which,  when  used 
in  conjunction  with  existing  Estimate  At  Completion  (EAC) 
techniques,  improve  the  srccess  of  existing  EAC  techniques. 
These  factors  attempted  to  account  for  changing  contract 
requirements  which  are  certain  to  occur  on  DoD  contracts . 
These  changing  requirements  often  result  in  a  need  to  add 
scope  and  budget  to  a  contract,  or  can  cause  contract  cost 
overruns  requiring  additional  budget. 

Improving  the  success  of  these  EAC  techniques  is 
increasingly  important  because  of  the  budget  cuts  being 
experienced  by  the  DoD  in  today's  changing  economic 
environment.  An  accurate  estimate  of  future  costs  is 
necessary  to  ensure  adequate  funding  levels  to  meet  DoD 
requirements . 

The  BAFs  were  constjnicted  by  compiling  actual  data  from 
534  DoD  contracts .  They  were  then  used  with  existing  EAC 
techniques  to  predict  final  contract  completion  costs . 
Descriptive  statistics  were  used  to  determine  if  the  BAFs 
improved  the  success  of  these  EAC  techniques . 


IX 


Unfortunately,  the  BAFs  did  not  increase  the  success  of 
the  SAC  techniques .  Although  these  particular  factors  did 
not  improve  the  success  of  these  EAC  techniques,  research  is 
still  needed  in  this  area.  Accurate  estimates  of  contract 
costs  is  still  important. 
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A  STUDY  TO  DETERMINE  IF  NEWLY  DEVELOPED  BUDGET  AT  COMPLETION 
(BAC)  ADJUSTMENT  FACTORS  IMPROVE  THE  SUCCESS  OF  PREVALENT 
INDEX-BASED  ESTIMATE  AT  COMPLETION  (EAC)  TECHNIQUES 


I.  Introduction 


General  Issue 

Department  of  Defense  (DoD)  contracts  for  new  weapon 
systems  frequently  experience  large  cost  overruns .  "Cost 
growth  in  weapons  has  been  a  serious  problem  for  the  past 
thirty  years"  (Weida,  1987:135).  Specifically,  one  source 
stated,  "the  average  cost  overrun  on  a  weapon  system  has 
been  around  40  percent"  (Gansler,  1989:4) .  Predicting  final 
costs  for  DoD  contracts  has  been  attempted  in  many  fashions, 
but  is  still  difficult. 

"Congress  plays  the  key  political  role  in  the  defense 
contracting  process  because  it  controls  the  purse  strings" 
(Mayer,  1991:11).  An  Air  Force  Times  staff  writer  quoted 
Representative  Ronald  V.  Dellums,  D-California,  whom  he 
referred  to  as  "the  liberal  House  Armed  Services  Committee 
chairman,  who  has  made  no  secret  of  his  belief  that  the 
nation  spends  too  much  on  defense,  as  saying,  'I  see  this 
budget  as  steady  funding  for  too  costly  programs'"  (Maze, 
1994:14).  The  DoD,  Congress,  and  the  American  Taxpayer  all 
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have  a  vested  interest  in  controlling  the  costs  of  future 
weapon  systems.  "Both  economic  and  political  factors 
combine  to  form  the  defense  budget”  (Weida,  1987:10) .  As 
the  economic  and  political  priorities  of  the  US  government 
shift,  a  growing  trend  within  the  DoD  is  to  extend  the 
acquisition  schedule  of  defense  systems. 

Stretching  out  a  program  means  extending 
research  and  production  over  more  years  than 
planned  in  order  to  make  them  affordable  each 
year.  The  down  side  is  that  overall  costs  can 
rise  when  fewer  planes  are  built  each  year, 
ultimately  driving  up  the  total  cost  and 
potentially  forcing  the  Air  Force  to  cut  the 
number  of  units  purchased.  (Watkins,  1994:30) 

Reducing  the  total  number  of  weapon  systems  purchased  has  an 

obvious  effect  upon  the  operational  use  and  strategy  of  a 

defense  system.  Thus,  the  ability  to  accurately  predict  the 

cost  impact  of  such  reductions  is  crucial  to  the  DoD,  as  it 

struggles  to  decide  which  weapon  systems  provide  the  most 

economical  means  of  defending  the  country. 

In  line  with  this  political  climate,  initial  cost 
estimates  for  new  weapon  systems  have  been  unrealistically 
low  in  an  attempt  to  obtain  Congressional  support  for  the 
program  (Gansler,  1989:8) .  This  practice  puts  program 
officials  in  a  difficult  management  position,  wherein,  they 
must  "accomplish  the  improbable  task  of  managing  [their] 
overspecified  and  underfunded  program  to  a  successful 
conclusion”  (Gansler,  1989:147) .  This  assertion  was  also 
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supported  by  another  author,  who  stated,  "Early  cost 
calculations  are  noteworthy  because  they  are  set  forth  with 
great  confidence,  are  backed  up  with  reams  of  analysis,  and 
are  always  wrong"  (Mayer,  1991:51) . 

The  OoD  has  been  aware  of  the  problems  associated  with 
contract  cost  overruns  for  a  number  of  years .  "The 
Department  of  Defense  has  initiated  a  number  of  major 
management  improvements  in  its  acquisition  process, 
specifically  aimed  at  overrun  reductions"  (Gansler, 
1978:268).  Improved  management  of  defense  acquisition 
programs  is  a  primary  method  for  controlling  program  costs . 
"Certainly  the  taxpayers '  willingness  to  support  a  strong 
defense  establishment  depends  in  large  measure  on  their 
perception  of  the  effectiveness  and  efficiency  with  which 
the  funds  are  spent"  (Gansler,  1989:141).  Savings  realized 
by  better  management  of  DoD  contracts  can  benefit  other 
programs  consistent  with  overall  government  fiscal  policy. 
Specifically,  cost  savings  on  Defense  contracts  can  be 
applied  to  other  areas  of  fiscal  interest .  As  the  United 
States  Government  budgeting  priorities  shift  from  Defense  to 
domestic  programs,  it  is  essential  that  the  DoD  avoid 
wasting  its  limited  funding.  Kenneth  R.  Mayer,  author  of 
The  Political  Economy  of  Defense  Contracting,  reported  that : 

Former  chairman  of  Council  of  Economic  Advisers 

Murray  Weidenbaum  summarized  the  major  trends  over 

the  past  fifty  years:  "Different  ways  of  gauging 
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resource  use  of  the  past  half  century  yield  the 
same  point:  defense  outlays  have  accounted  for  a 
declining  share  of  GNP;  defense  spending  has  been 
a  declining  portion  of  the  federal  budget;  defense 
manpower  has  represented  a  declining  fraction  of 
the  nation's  workforce:  defense  has  received  a 
diminishing  portion  of  the  nation's  research  and 
development  funding  .  .  .  over  a  very  significant 
time  period,  military  activities  have  been  a 
steadily  smaller  factor  in  the  American  Economy. 

(Mayer,  1991:268) 

The  DoD  must  get  the  most  out  of  its  future  weapon  systems ' 
development  and  procurement  budgets.  "The  ever-shrinking 
defense  budget  continues  to  press  the  service  into 
considering  more  programs  for  termination  or  drastic 
restructuring"  (Watkins,  1994:34).  In  light  of  this,  final 
contract  cost  predictions  are  becoming  even  more  crucial . 


The  Navy's  A-12  program  is  a  prime  example  of  what  can 
happen  when  program  management  does  not  accurately  or  timely 
predict  and  report  final  contract  costs.  Among  other 
reasons,  the  A-12  program  was  said  to  be  canceled  because 
large  discrepancies  were  discovered  in  its  estimated 
completion  cost  (Morrison,  1991:30).  For  DoD  contracts, 
both  the  contractor  and  the  government  program  office 
develop  an  estimate  of  final  contract  cost,  known  as  an 
Estimate  at  Completion  (EAC) ,  Specifically,  when  problems 
arise,  an  EAC  is  required  of  the  contractor,  known  as  the 
contractor's  Latest  Revised  Estimate  (LRE) .  Additionally, 
the  government  program  office  often  calculates  independent 
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EACs  to  test  the  reasonableness  of  the  contractor's  LRE 
(Christensen,  1992:6). 

The  EAC  can  be  compared  to  the  funds  available  to 
accomplish  the  contract,  known  as  the  budget  at  completion 
(BAG) ,  to  determine  if  the  contract  can  be  completed  within 
allocated  funds .  If  the  EAC  exceeds  the  BAG,  program 
management  needs  to  take  necessary  action  to  ensure  contract 
completion.  If  possible,  program  management  might  seek 
additional  funding  or  reduce  contract  requirements  and 
constraints.  Based  on  potential  cost  overruns,  an  accurate 
and  timely  technique  is  needed  to  inform  program  management 
when  an  adjustment  to  technical  or  program  requirements  is 
necessary.  Prompt  and  accurate  management  decisions  can 
avoid  wasting  government  funding  and  potentially  save  a 
program.  "By  Public  Law  (Nunn-McCurdy  Amendment) ,  Congress 
must  consider  canceling  defense  contracts  when  they  exceed 
certain  unit-cost  thresholds"  (Christensen,  1992:5). 

Specific  Problem 

The  purpose  of  this  research  was  to  determine  whether 
newly  developed  BAG  adjustment  factors  improved  the  success 
of  existing  EAC  techniques  in  predicting  the  final  cost  of 
OoD  weapon  system  contracts . 
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Research  Scope  and  Qbjectivaa. 

Specific  research  objectives  for  this  study  were: 

1.  Determine  the  EAC  methods  currently  being  used  and 
believed  to  be  successful. 

2 .  Determine  the  work  done  in  the  area  of  estimates  at 
completion,  specifically,  index-based  (linear)  methods. 

3 .  Determine  the  scoring  methodologies  that  have  been  used 
to  rate  the  success  of  EAC  forecasting  techniques . 

4.  Use  historical  data  to  develop  BAG  adjustment  factors, 
which  are  used  in  conjunction  with  current  EAC  methods,  to 
predict  final  contract  costs  at  various  points  in  the  life 
of  DoD  contracts . 

5.  Determine  if  the  BAG  adjustment  factors  developed  in 
research  objective  four  improved  the  success  of  the 
prevalent  EAC  methods  determined  in  research  objective  one. 

Historical  contract  cost  performance  report  (CPR)  data 
contained  in  the  Defense  Acquisition  Executive  Summary 
(DAES)  database  were  used  for  this  study.  There  were  some 
limitations  using  this  database,  such  as,  the  origin  of  the 
information.  The  data  within  the  DAES  database  came  from 
contractor  financial  reports  submitted  to  the  program 
office.  These  reports  were  generated  by  contractor 
performance  measurement  systems  that  were  supposed  to  be 
validated  by  the  government  to  ensure  the  accuracy  and 
reliaUaility  of  the  data.  If  the  government  mistakenly 
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validated  a  perfoinnance  measurement  system  that  was  not 
valid  or  contained  bias,  the  data  received  in  the  financial 
reports  may  not  have  been  indicative  of  the  contractor's 
performance.  For  the  purposes  of  this  research,  the 
assumption  was  made  that  the  data  were  from  properly 
validated  systems  and  were  accurately  reported. 

Another  limitation  of  the  DAES  database  was  it  did  not 
indicate  the  method  used  by  the  program  office  to  develop 
its  EAC.  As  a  result,  conclusions  were  limited  to  general 
statements  of  how  the  newly  developed  BAG  adjustment  factors 
might  help  better  predict  contract  completion  costs . 
Conclusions  did  not  include  statements  concerning  specific 
EAC  techniques  used  by  the  system  program  office  (SPO) .  A 
third  limitation  was  that  the  data  in  the  DAES  Database  did 
not  include  contract  cost  information  through  the  one 
hundred  percent  completion  point.  In  most  cases,  the  data 
only  included  information  through  the  80%  or  90%  completion 
point.  For  this  reason,  it  was  necessary  to  extrapolate 
contract  costs  to  the  100%  completion  point  to  establish  the 
actual  completion  costs  used  to  measure  the  EAC  techniques ' 
success . 

This  chapter  has  introduced  the  general  problem 
associated  with  contract  cost  overruns  facing  program 
management  within  the  Defense  Industry  today.  It  has  also 
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highlighted  how  better  estimates  at  completion  for  DoD 
contracts  can  aid  top  defense  and  other  national  leaders  in 
their  attempts  to  restructure  the  U.S.  defense  budget. 
Chapter  Two  is  a  review  of  the  most  notable  literature  on 
Estimate  at  Completion  research.  Chapter  Two  identifies  the 
most  prevalent  EAC  techniques  to  be  tested  in  the  analysis 
portion  of  this  research.  Chapter  Three  describes  the 
methodology  used  to  compare  the  success  of  the  different  EAC 
techniques.  Chapter  Four  discusses  the  analysis  and 
provides  the  results  of  the  EAC  technique  comparison. 
Finally,  Chapter  Five  lists  general  conclusions  and  findings 
on  EAC  technique  success  based  upon  the  comparative  analysis 
completed  and  described  in  Chapters  Three  and  Four. 
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The  primary  purposes  of  reviewing  other  literature  were 
to  avoid  duplicating  EAC  work  already  accomplished  and  to 
discover  what  areas  of  EAC  research  and  methodology  needed 
further  attention.  In  addition,  the  review  of  other  EAC 
literature  helped  determine  which  EAC  methods  have  been 
considered  successful.  Hence,  the  literature  review 
assisted  in  scoping  the  comparative  portion  of  this  analysis 
to  those  EAC  techniques  which  have  previously  been 
recommended.  For  a  more  detailed  review  of  these  EAC 
studies,  see  Christensen,  et  al. 

Before  discussing  the  findings  of  the  literature 
review,  a  brief  description  of  the  terminology  that  will  be 
used  is  in  order.  A  series  of  definitions  are  important  to 
understand  the  process  of  computing  an  Estimate  at 
Completion  (EAC) .  The  following  narrative  outlines 
applicable  terms . 

First  of  all,  an  EAC  consists  of  the  current  costs 
incurred  on  a  contract,  plus  an  estimate  of  the  cost  of  work 
remaining  to  complete  the  contract.  Typically,  contractors 
submit  a  similar  figure,  called  the  Latest  Revised  Estimate 
(LRE) ,  which  includes  their  own  estimate  of  the  cost  of  work 


remaining  to  complete  the  contract.  The  current  costs 
incurred  on  the  contract  are  referred  to  as  the  Actual  Cost 
of  Work  Performed  (ACWP),  while  the  work  remaining  is  called 
the  Budgeted  Cost  of  Work  Remaining  (BCWR) .  The  budget 
identified  for  the  known  work  on  a  contract  is  referred  to 
as  Budget  at  Completion  (BAC) .  The  BCWR  is  defined  as  the 
BAC  minus  the  budgeted  cost  of  work  performed  (BCWP)  to 
date. 


The  Cost  Performance  Index  (CPI)  and  the  Schedule 
Performance  Index  (SPI)  are  commonly  used  in  calculating  an 
EAC.  The  CPI  is  the  ratio  of  the  BCWP  to  the  ACWP,  while 
the  SPI  is  the  ratio  of  the  BCWP  to  the  planned  cost  of  work 
scheduled.  The  planned  cost  of  work  scheduled  is  called  the 
Budgeted  Cost  for  Work  Scheduled  (BCWS) .  For  the  purposes 
of  this  study,  both  indexes  will  be  computed  from  cumulative 
data. 


_  BCWPc«. 
ACWPc* 


(2.1) 


_  BCWPcu, 
BCWSc« 


(2.2) 


There  are  a  number  of  other  terms  pertinent  to  this 
study.  The  contract  Cost  at  Completion  (CAC)  is  the  actual 
cost  of  the  contract  at  its  close.  In  addition,  chapter  14 
of  Air  Force  Material  Command  Pamphlet  (AFMCP)  173-4  defines 
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other  applicable  terms  in  the  following  manner  (AFMCP  173-4, 
1992:108) .  The  time-phased  cumulative  total  of  all  work 
within  a  contract  is  referred  to  as  the  Performance 
Measurement  Baseline  (PMB) .  The  amount  of  total  allocated 
budget  withheld  for  management  control  purposes  is  called 
the  Management  Reserve  (MR) .  The  Contract  Budget  Baseline 
(CBB)  is  the  negotiated  contract  cost  plus  the  estimated 
cost  of  authorized  unpriced  work.  The  Over  Target  Baseline 
(0TB)  is  a  PMB  resulting  from  formal  reprogramming  by  the 
contractor,  with  customer  approval,  which  establishes  budget 
allocations  in  excess  of  the  CBB.  The  Total  Allocated 
Budget  (TAB)  is  the  sum  of  all  allocated  budgets .  The 
following  formulas  will  help  describe  the  relationship 
between  the  PMB,  MR,  0TB,  and  the  TAB. 

CBB  =  BAC  Of  the  PMB  +  MR  (2.3a) 

TAB  =  CBB  +  0TB  (2.3b) 

Having  stated  these  definition,  the  remainder  of  the 
literature  review  can  proceed. 

A  logical  starting  point  for  the  literature  review  was 
to  identify  those  EAC  forecasting  techniques  that  were 
currently  being  used.  Many  weapon  system  acquisition 
programs  in  the  United  States  Air  Force  use  the  Performance 
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Analyzer  (PA)  (Version  4.0)  Software  "to  streamline  the 
analysis  and  reporting  process  associated  with  Cost 
Performance  Reports  (CPR) ,  Cost/Schedule  Status  Reports 
(C/SSR)  and  Contract  Funds  Status  Reports  (CFSR) "  (Software 
User's  Manual  for  the  Performance  Analyzer.  1991 :i).  One  of 
the  capabilities  of  this  software  is  allowing  the  cost 
analyst  in  a  System  Program  Directorate  (SPD)  to  compute  and 
display  an  EAC  at  the  touch  of  a  key  on  his  computer 
keyboard.  Historically,  the  cost  analyst  could  choose  from 
a  list  of  about  ten  different  EAC  techniques .  The  list  of 
possible  EAC  techniques  has  changed  over  the  years  as 
subsequent  versions  of  the  PA  Software  have  been  released. 
The  most  current  version.  Version  4.0,  was  released  on  24 
June  1994.  As  with  any  software  product,  feedback  from  its 
users  is  considered  before  the  release  of  subsequent 
versions.  Therefore,  the  list  of  forecasting  techniques  in 
PA  represented  a  sample  of  EAC  methodologies  currently  being 
used  in  Air  Force,  Army,  and  Navy  SPDs . 

Starting  with  this  list  of  ElAC  techniques,  the 
literature  review  focused  on  studies  which  tested  the 
success  of  certain  EAC  techniques  available  in  PA.  Since 
the  intent  of  this  study  was  to  compare  the  success  of 
current  EAC  methods  used  with,  and  without,  newly  developed 
BAC  adjustment  factors,  the  focus  of  the  literature  review 
was  also  on  techniques  that  could  be  computed  using  data  in 
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the  DAES  database.  The  DAES  database  only  contains 
cumulative  contract  performance  data  at  sporadic  intervals . 
That  is,  the  DAES  data  were  not  reported  at  regular  monthly, 
quarterly,  or  annual  intervals.  For  these  reasons,  the 
literature  review  centered  on  the  following  PA  forecasting 
techniques : 


Cumulative  Cost  Performance  Index  (CPI) 


EACl 


ACWPcum  + 


BAC  —  BCWPcum 
CPI 


(2.4) 


Weighted  Coat  and  Schedule 


EAC2 


ACWPcum 


BAC  —  BCWPcum 
A(CPI)  +  B(SPI) 


where:  A  =  the  cost  weighting 

B  =  the  schedule  weighting 
A  +  B  must  equal  1 


(2.5) 


The  most  common  weighting  scheme  encountered  during  the 
literature  review  was  0.8  for  cost  and  0.2  for  schedule. 
Therefore,  this  was  the  weighting  scheme  used  in  the 
analysis  portion  of  this  study  as  well. 


NAVSEA  90 '3  Formula 

EAC3  -  ACWPcum  +  CBB  +  0TB  -  BCWPcum  + 


(2.6) 


(CBB  +  0TB  -  BCWPcum) 


CCPI 


-  CBB  -  0TB  +  BCWPcum 


BCWPcum 
CBB  +  0TB 
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where : 


CCPI 


BCWPcum  V  ^  BCWPow  y  L  ^  BCWSciffi  -  BCWScum 

2*ACWPcumJ  I  BCWScum  j  I  CBB  +  0TB  J 


These  first  three  index-based  EAC  techniques,  along 
with  a  fourth  technique,  the  Schedule  Cost  Index  (SCI) ,  were 
included  in  this  study.  Even  though  it  is  not  directly 
available  within  Performance  Analyzer,  the  SCI  was  evaluated 
with  positive  results  in  three  of  the  studies  reviewed; 
therefore,  it  was  included  in  this  study. 


SCHEDULE  COST  INDEX  (SCI)  EAC 


EAC4 


ACWPcum 


^  BAC  —  BCWPctw 
CPI*  SPI 


(2.7) 


The  reviewed  literature  was  summarized,  and  arranged  in 
chronological  order.  Then,  conclusions  were  drawn;  and  the 
EAC  methods  that  seemed  to  perform  well,  as  well  as  the 
scoring  techniques  used  to  evaluate  their  success,  were 
identified.  Results  are  summarized  in  Table  2-1.  The  left 
hand  colxomn  of  Table  2-1  lists  the  authors  of  the  five  EAC 
comparative  studies  that  were  reviewed.  In  cases  where 
there  were  more  than  one  author,  only  the  name  of  the  first 
author  was  listed.  See  the  Bibliography  for  a  complete 
reference.  The  top  row  of  Table  2-1  lists  the  EAC 
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forecasting  techniques  that  were  selected  for  inclusion  in 
this  study.  Reading  across  the  rows  and  down  the  columns, 
to  the  intersection  of  an  author  and  a  technique,  there  is  a 
brief  explanation  as  to  how  well  that  particular  technique 
performed  within  that  author's  study.  An  N/A  means  that 
particular  EAC  technique  was  not  included  in  that  author's 
study.  As  can  be  seen  from  the  table,  conclusions  on  the 
success  of  the  EAC  techniques  varied.  The  last  column  of 
Table  2-1  lists  the  scoring  methodology  used  within  each 
study.  A  more  detailed  discussion  of  the  results  and  the 
scoring  methodologies  can  be  found  in  the  following 
paragraphs  on  each  study. 

Land  and  Preston 

A  thesis  accomplished  for  the  Air  Force  Institute  of 
Technology,  Air  University,  compared  two  cost  forecasting 
models .  The  models  compared  were  The  Automated  Financial 
Analysis  Program,  Electronic  Systems  Division  (ESD) , 

November  1976,  and  A  Cost  Performance  Forecasting  Concept 
and  Model,  Aeronautical  Systems  Division  (ASD) ,  November 
1974  (Land  and  Preston,  1980:2) .  The  ESD  model  consisted  of 
six  index  based  methods,  while  the  ASD  model  included  two 
non-linear  regression  based  methods  for  forecasting  the  cost 
at  completion  for  various  DoD  programs  (Land  and  Preston, 
1980:6)  .  The  comparison  was  accomplished  using  actual  cost 
data  from  25  ASD  programs,  plus  5  aircraft  programs,  all 
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with  periods  of  performance  greater  than  12  months  (Land  and 
Preston,  1980:20). 

EAC  technique  performance  was  measured  using  the  Mean 
Absolute  Percentage  Error  (MAPE)  (Land  and  Preston, 

1980:32) .  The  MAPE  was  used  because  percentages  eliminated 
the  impact  of  dollar  magnitude  difference  between  contracts . 
The  test  hypothesis  was  that  the  non-linear  methods  were 
more  accurate  than  the  linear  index  based  methods.  A  single 
factor  analysis  of  variance  (ANOVA)  was  used  to  test  if  the 
MAPE  from  one  method  was  significantly  different  than  the 
others .  If  there  was  no  significant  difference  between 
MME's,  then  the  test  hypothesis  was  not  supported. 
Additionally,  if  one  or  more  MAPE's  tested  significantly 
different  from  the  others,  Fisher's  least  significant 
difference  (LSD)  test  was  applied  to  determine  the  true 
differences  between  MAPE's.  The  methods  were  evaluated  over 
various  types  of  contracts  at  different  percents  complete. 
The  study  concluded  that  the  hypothesis  that  non-linear 
methods  are  more  accurate  than  linear  models  at  forecasting 
costs  at  completion  could  not  be  supported  (Land  and 
Preston,  1980:53).  The  Land  and  Preston  conclusion  is 
applicable  to  this  study  because  this  study  will  only  be 
reviewing  linear  methods  for  estimating  costs  at  completion. 
Additionally,  the  CPI (Cum)  MAPE  proved  slightly  lower  than 
the  other  methods  tested  (Land  and  Preston,  1980:40).  This 
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further  supports  the  inclusion  of  the  CPI (Cum)  EAC  technique 
in  this  study . 


Covach.  et  al 

The  Mantech  report  consisted  of  testing  the  accuracy, 
timeliness,  and  stability  of  12  index-based,  and  12 
regression-based  EAC  techniques  on  27  Navy  contracts . 

Methods  based  on  manpower  constnnption  and  lower  level  Work 
Breakdown  Structure  (WBS)  elements  were  also  evaluated.  The 
study  concluded  that  certain  index-based  methods  were  more 
accurate  than  others  depending  on  the  stage  of  the  program's 
life  (early,  middle,  or  late) . 


The  scoring  technique  used  in  the  Mantech  report 
evaluated  an  EAC  technique's  performance  based  upon  four 
subjectively  established  evaluation  questions .  These 
questions  are  as  follows : 

1.  How  many  months  out  of  the  contract  life  would 
each  method  produce  an  EAC  that  is  within 
plus/minus  10%  of  the  CAC? 

2 .  How  many  months  would  each  method ' s  EAC  be 
closer  to  the  CAC  than  the  Budget  at  Completion 
(BAC)  is? 

3 .  How  many  months  in  the  contract  life  will  a 
method's  EAC  be  superior  to  the  contractor's  LRE? 

4 .  If  the  contract  life  is  divided  into  fifths 
(quintiles)  based  on  budgeted  dollars,  how  would 
the  various  methods  behave  at  the  various  stages 
of  the  contract?  (Covach,  1981:22) 

A  numerical  grade  was  computed  for  each  method  by  awarding  a 

+1.0  to  each  success  and  a  -1.0  for  each  failure  in  its 
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ability  to  be  within  plus /minus  10%  of  the  CAC,  or 
outperform  the  BAG,  or  LRE.  This  was  also  accomplished  for 
the  various  stages  of  the  contract  (i.e.  quintiles) .  Based 
upon  this  numerical  grade  subjective  recommendations  were 
made  for  use  of  different  EAC  techniques  depending  upon  the 
stage  of  the  contract.  The  Mantech  report  recommended 
CPI (Cum)  and  SCI  for  contracts  in  the  0-60%  complete  range. 
Two  limitations  of  the  study  were  the  relatively  small  data 
sample  (only  27  contracts)  and  the  fact  that  index-based  and 
regression-based  methods  were  not  compared  to  each  other. 

No  specific  conclusions  were  made  concerning  the  manpower 
techniques;  and  lower  level  analysis  was  "not  significantly 
more  accurate  than  estimates  based  on  aggregate  budget 
dollars"  (Covach,  1981:3) . 

Bright  and  Howard 

A  paper  written  for  the  Cost  Analysis  Division,  US  Army 
Missile  Command,  compared  EAC  forecasting  techniques  that 
were  employed  in  their  Automated  Contractor  Performance 
Measurement  System  (ACPMS)  (Bright  and  Howard,  1981:2-3). 
Their  analysis  included  nine  index  based  and  two  regression 
based  EAC  techniques  on  eleven  Army  contracts.  Bright  and 
Howard  recommended  certain  techniques  over  others  depending 
on  the  completion  stage  of  the  contract  (0-30%,  31-80%,  or 
81-100%  complete) .  They  also  concluded  that  "CPI  X  SPI  (or 
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SCI)  appeared  to  provide  better  overall  forecasts  than  the 
other  methods"  (Bright  and  Howard,  1981:19). 

The  scoring  technique  used  in  the  Bright  and  Howard 
paper  included  two  approaches:  graphic  and  rating.  The 
graphic  approach  consisted  of  plotting  average  percent 
errors  of  the  different  EAC  techniques  versus  percent 
complete  of  the  contract.  Percent  errors  are  calculated  by 
dividing  the  error  (EAC-CAC)  by  the  final  contract  cost 
(CAC) .  Only  selected  example  graphs  were  provided  in  the 
report  making  relative  comparisons  based  on  the  graphic 
approach  of  all  the  EAC  techniques  studied  impossible.  The 
second  approach  was  a  scoring  system,  similar  to  the  Mantech 
scoring  technique,  that  gave  one  point  to  the  forecast  that 
performed  the  best  for  each  month.  That  is,  the  EAC 
technique  that  had  the  lowest  percent  error.  Fractions  of 
points  were  given  to  other  forecasts  depending  on  how  close 
they  were  to  the  best  technique  for  that  particular  month. 
The  rules  for  assigning  fractions  of  points  were  not 
provided.  In  addition,  the  overall  scores  for  each 
technique  using  the  rating  approach  were  not  provided  as 
back-up  to  support  the  final  conclusions  and 
recommendations.  The  database  used  in  Bright  and  Howard's 
paper  included  eleven  Army  Missile  Command  (MICOM) 
contracts . 
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An  additional  benefit  in  reviewing  Bright  and  Howard's 


paper  was  that  it  included  discussion  on  the  treatment  of 
budget  being  added  to  contracts  when  additional  scope  is 
added  to  the  contract,  and  when  the  contract  is  rebaselined 
due  to  cost  overruns .  They  used  an  approach  in  their 
analysis  of  adjusting  the  rebaselined  data  to  be  consistent 
with  the  budget  prior  to  rebaselining.  In  order  to  use  an 
index-based  forecasting  technique,  it  is  necessary  to  obtain 
a  dollar  figure  which  represents  the  work  remaining  to  be 
done  on  the  contract.  This  dollar  figure  is  rather  easy  to 
compute  at  any  given  point  of  the  contract.  The  work 
remaining  to  be  accomplished  on  a  contract,  or  Budgeted  Cost 
of  Work  Remaining  (BCWR)  is  normally  calculated  as  follows : 
BCWR  =-  BAG  -  BCWP .  However,  if  after  that  particular 
point,  the  contract  is  rebaselined  and/or  scope  is  added, 
the  BAG  used  to  compute  BCWR  was  understated.  The  result  of 
using  this  understated  BCWR  when  forecasting  the  actual 
contract  completion  costs  was  an  understated  forecast,  or 
EAC. 


Therefore,  Bright  and  Howard  attempted  to  obtain  a 
better  dollar  figure  for  the  contract  work  remaining  as 
follows :  "when  computing  work  remaining  the  final  BAG  was 

used  rather  than  the  BAG  immediately  following  the 
rebaseline  and  an  adjusted  final  BAG  was  used  in  place  of 
the  BAG  immediately  prior  to  rebaselining"  (Bright  and 
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Howard,  1981:11).  In  other  words,  when  applying  a 
forecasting  technique  on  data  subsequent  to  a  rebaseline, 
the  BCWR  was  obtained  by  subtracting  cumulative  BCWP  from 
the  final  contract  BAG.  For  example,  given  that  total  BAG 
after  a  rebaseline  was  $100  and  cumulative  BGWP  was  $50,  if 
the  final  BAG  was  $150,  then  BGWR  was  computed  to  be  $100, 
($150  -  $50) .  When  applying  a  forecasting  technique  on  data 
accumulated  prior  to  a  rebaseline,  the  BGWR  was  obtained  by 
subtracting  cumulative  BGWP  from  an  adjusted  final  BAG. 

Using  the  same  data  from  the  previous  example,  if  the  BAG 
prior  to  the  rebaseline  was  $70  and  the  cvimulative  BGWP  was 
$45,  then  a  BGWR  of  $30  was  computed  as  follows: 


BGWR 


FinalBAG* 


(BAGprior 

to  rebasallno  “  BGWPprlor ...) 
(BAGrapasailned  “  BGWPrebasellned  )  J 


BGWPprlor...  (2.8) 


BGWR  » 


$  150* 


($70  -  $45) 
($  100  -  $50) 


-  $45 


The  result  of  using  this  adjusted  final  BAG  was  that  $75  was 
used  as  the  BAG  at  that  particular  point  in  the  contract 
instead  of  the  true  BAG  of  $70.  In  both  examples  (prior  and 
subsequent  to  rebaselines) ,  the  increased  BAG  that  was  used 
accounted  for  the  additional  budget  that  was  eventually 
added  to  the  contract . 


In  addition,  the  final  cost  that  Bright  and  Howard 
compared  with  their  forecasts  was  also  adjusted  to  account 
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for  scope  being  added  after  a  particular  forecast  was  made. 
Their  final  cost  was  adjusted  as  follows: 


Final  Cost 


BAG  (time  of  forecast) 
BAG  (final) 


*  Actual  Final  Cost 


(2.9) 


These  data  adjustment  techniques  were  easy  to  apply  on 
contract  coat  information  from  completed  contracts . 

However,  coat  analysts  who  are  trying  to  forecast  an  BAG  for 
their  program  can  not  use  this  method,  because  they  have  no 
idea  of  what  the  BAG  (Final)  is  going  to  be. 

The  problem  of  not  adjusting  the  data  for  reasons  of 
rebaselines  and  added  scope  is  that  the  forecasting 
techniques  currently  being  used,  especially  the  index-based 
ones  which  include  the  BCWR  in  their  formulas,  are  not 
capable  of  predicting  when  budget  will  be  added  to  contracts 
for  scope  additions  and  rebaselines .  By  adjusting  the 
historical  data,  as  was  accomplished  in  the  Bright  and 
Howard  thesis,  the  EAGs  would  only  appear  more  accurate. 

When  in  fact,  what  is  needed  is  a  forecasting  technique 
which  can  account  for  these  types  of  budget  additions .  The 
uncertainty  of  budget  being  added  to  contracts  for  reasons 
of  additional  scope,  and  rebaselines  due  to  cost  overruns  is 
an  important  component  of  the  proposed  BAG  adjustment 
factors  to  be  developed  in  this  research. 
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The  Price  thesis  examined  how  well  five  index-based  EAC 
techniques  and  one  regression-based  EAC  technique  performed 
in  estimating  a  program's  cost  at  completion.  It  also 
tested  the  statistical  equivalence  of  the  different  EAC 
methodologies .  The  index  based  EAC  techniques  included 
current  month  CPI,  three  month  CPI,  cumulative  CPI,  a 
variant  of  the  weighted  cost  and  schedule  (see  note  2  in 
Table  2-1) ,  and  a  fifth  method  defined  in  the  following 
manner : 

EAC  =  EACC  +  ETCS  (2.10) 

where:  ElACC  =  [.12  *  EAC  (current  month  CPI)  + 

.24  *  EAC (three  month  CPI)  + 

.64  *  EAC (cumulative  CPI)] 

ETCS  =  (months  behind  schedule)  *  ACWP^ate  *  (^^S) 

ACWPrate  =  ACWPcum  /  TNUM 

TNUM  =»  number  of  months  from  beginning  of  contract 
to  latest  CPR 

This  fifth  index  based  EAC  technique  was  rather  involved. 

The  regression  based  technique  was  based  on  ACWP  regression. 
It  used  a  least-squares-best-fit  on  ACWP  to  establish  a 
trend  line  which  was  extended  to  the  zero  work  remaining 
point  to  become  the  EAC  (Price,  1985:11).  The  conclusion  of 
the  thesis  was  that  two  of  the  index-based  methods  were 
selected  as  the  best  predictors.  "It  is  clear  .  .  .  that 
the  WEIGHTED  CPI/SPI  and  the  CUMULATIVE  CPI  are  closely 
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matched  in  their  respective  predictive  power”  (Price, 
1985:32) . 

The  scoring  technique  used  in  the  Price  thesis 
consisted  of  formulating  a  univariate  (a  single  dependent 
varicUsle)  regression  line  based  on  the  CAC  (dependent 
variable)  and  the  EAC  generated  by  each  technique 
(independent  variable) ,  and  computing  the  coefficient  of 
determination,  R-square.  The  highest  R-squares  were  deemed 
the  best  estimators .  It  was  also  necessary  for  Price  to 
make  certain  assumptions  when  using  linear  regression. 

These  assumptions  were  as  follows: 

1.  For  any  fixed  value  of  X,  Y  is  a  random 
variable  with  a  certain  probability  distribution. 

2 .  The  mean  value  of  Y  .  .  .  is  a  straight  line 
function  of  X. 

3.  The  variance  of  Y  is  the  same  for  any  X.  This 
assumption  is  called  the  assumption  of 
homoscedasticity . 

4 .  For  any  fixed  value  of  X,  Y  has  a  normal 
distribution.  (Price,  1985:19-20) 

These  assumptions  were  successfully  tested,  except  for  the 

assumption  of  homoscedasticity.  In  that  case.  Price  assumed 

that  "while  the  data  did  exhibit  heteroscedastic  tendencies 

they  did  not  severely  degrade  the  analysis"  (Price, 

1985:21) . 

The  database  included  57  on-going  research  and 
development  programs.  A  short-coming  of  the  thesis  was 
that,  since  the  programs  were  still  on-going,  the  latest 
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cumulative  ACWP  and  BCWP  were  used  as  estimates  for  the 
final  contract  cost  at  completion  (CAC)  and  budget  at 
completion  (BAG)  .  The  ElAC  techniques  were  then  used  to 
predict  what  the  coats  would  be  for  the  work  completed  to 
date. 

Eledal  .and  Chance 

A  paper  written  for  the  Directorate  of  Cost, 
Aeronautical  Systems  Division,  compared  the  success  of  seven 
index-based  EAC  formulas  on  18  Aeronautical  Systems  Division 
(ASD)  programs  (Riedel  and  Chance,  1989:4-6) .  They  chose 
the  seven  formulas  in  their  analysis  after  reviewing  the 
results  from  several  data  sources  they  had  gathered:  an  Air 
Force  Institute  of  Technology  (AFIT)  Thesis,  a  Defense 
Systems  Management  College  (DSMC)  Course,  an  AFIT  Short 
Course  Text,  and  an  article  from  Program  Manager.  Their 
study  only  included  EAC  formulas  that  were  recommended  for 
use  by  the  reviewed  data  sources  (Riedel  and  Chance, 

1989 :Ch  1).  Riedel  and  Chance  concluded  that  certain  EAC 
methods  performed  better  than  others  depending  upon  the 
completion  stage  and  type  of  contract  (research  and 
development  or  production  contracts) . 

The  scoring  techniqpie  used  by  Riedel  and  Chance 
consisted  of  two  methods.  "The  first,  was  based  on  the 
average  absolute  value  of  the  deviation  percentages  .  .  . 
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the  second  was  based  on  the  average  rank  order  of  the 
deviation  percentages"  (Riedel  and  Chance,  1989:18) .  Their 
average  absolute  value  of  the  deviation  percentages  was 
essentially  a  comparison  of  the  Mean  Absolute  Percentage 
Errors  (MAPE) .  After  EACs  are  calculated  using  the 
different  techniques,  the  absolute  value  of  the  difference 
(error)  between  the  EAC  and  the  CAC  is  summed  and  divided  by 
the  number  of  EACs  generated  to  arrive  at  a  MAPE .  The 
average  rank  order  of  the  deviation  percentages  was 
essentially  a  cross-check  of  the  first  method.  Instead  of 
looking  at  the  raw  MAPE  values,  the  EACs'  MAPEs  were  rank 
ordered  (lowest  =  1,  etc.).  The  average  rank  order  for  each 
method  was  calculated  by  summing  the  EAC  technique's  rank 
order  for  all  programs  and  dividing  by  the  number  of 
programs .  Those  EAC  methods  with  the  lowest  percentage 
deviation  and  percentage  deviation  rankings  (computed  by 
averaging  all  the  programs)  at  various  stages  of  contract 
completion  were  deemed  the  best  EAC  techniques  for  those 
stages.  Those  EAC  methods  with  the  lowest  overall  average 
deviation  and  overall  average  deviation  rankings  (computed 
by  averaging  the  percentage  deviations  at  each  contract 
stage)  were  deemed  the  beat  overall  EAC  techniques.  They 
admitted  in  their  recommendations  that  the  database  used  in 
certain  facets  of  their  analysis  was  relatively  small 
(Riedel  and  Chance,  1989:80).  Only  eight  aircraft,  five 
avionics,  and  five  engine  programs  were  analyzed. 
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Qtiner  eac  Reiatea  papers 

The  following  papers,  studies,  etc.  were  reviewed  to 
see  if  any  further  EAC  research  had  been  conducted  since  the 
publication  of  the  previously  reviewed  studies.  The  review 
of  these  other  EAC  related  papers  would  eliminate  any 
possible  duplication  of  effort  by  this  research  study. 

Major  David  S.  Christensen,  et  al,  published  a  paper 
comparing  25  EAC  studies  (Christensen,  et  al,  1992:208). 

The  ELAC  studies  were  categorized  into  one  of  the  following 
categories:  index  based,  regression  and  other.  The  index 
based  methods  included  various  combinations  of  the  budgeted 
and  actual  costs  of  work  performed.  Indexes  were  used  to 
adjust  the  cost  of  work  still  to  be  completed  on  the 
contract .  Regression  based  models  reviewed  included  both 
linear  and  non-linear  relationships  between  two  variables, 
typically  budgeted  and  actual  cost  of  work  performed.  The 
final  category,  "other, "  was  used  to  include  all  models  that 
did  not  fall  into  the  first  two  categories . 

Christensen's  review  produced  two  generalizations. 
First,  the  "accuracy  of  regression-based  models  over  index- 
based  formulas  has  not  been  established"  with  respect  to 
work  that  has  currently  been  accomplished  (Christensen,  et 
al,  1992:220).  Results  are  inconclusive  as  to  which  method 
is  superior.  And  second,  the  "accuracy  of  index-based 
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formulas  depends  on  the  type  of  system,  and  the  stage  and 
phase  of  the  contract”  (Christensen,  et  al,  1992:221). 
Specifically,  no  one  formula  is  best  in  all  situations. 

An  article  presented  in  the  National  Contract 
Management  Journal,  evaluated  the  stability  of  the  Cost 
Performance  Index  (CPI)  (Christensen  and  Heise,  1993:7). 

The  CPI  is  an  index  used  to  analyze  cost  and  schedule 
performance  data  reported  by  defense  contractors .  The  study 
evaluated  the  CPI  using  cost  data  from  155  contracts  from  44 
different  programs  (Christensen  and  Heise,  1993:9).  The 
study  concluded  that  "the  cumulative  CPI  was  stable  from  the 
20  percent  completion  point  with  95  percent  confidence. 
Stability  was  defined  in  terms  of  CPI  range  being  less  than 
.200"  (Christensen  and  Heise,  1993:13) .  This  finding  was 
significant  because  it  showed  that  the  government  could 
conclude  with  some  confidence  that  a  contract  was  in  trouble 
when  the  contract  overran  the  budget  at  the  20%  completion 
point.  The  cumulative  CPI  stability  is  important  to  this 
thesis  effort  because  cximulative  CPI  is  one  of  the 
techniques  that  will  be  evaluated. 

Excessive  optimism  in  cost  overrun  projections  was 
highlighted  in  a  study  published  in  the  Acquisition  Review 
Quarterly  (Christensen,  1994:25).  The  study  examined  cost 
overrun  data  from  64  completed  acquisition  contracts.  The 
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study  evaluated  the  overruns  through  various  contract  types, 
phases,  types  of  weapon  systems  and  military  service 
responsible  (Christensen,  1994:32).  On  a  summary  level,  the 
government  estimate  was  lower  than  the  actual  overrun,  and 
the  contractor  estimate  was  even  lower  still.  The  study 
concluded  that  both  the  government  and  contractor  estimates 
were  significantly  lower  than  actual  cost  overruns .  This 
conclusion  shows  the  need  for  more  realistic  and  more 
accurate  estimating  techniques. 

Conclusions 

The  literature  review  revealed  that  no  single  EAC 
method  outperformed  all  other  techniques .  Much  of  the  past 
research  recommended  different  EAC  techniques.  Many  of 
their  recommendations  suggested  using  different  EAC 
techniques  at  different  times  in  a  programs  life-cycle 
(early,  middle,  or  late) .  Additionally,  the  literature 
review  identified  certain  index-based  EAC  techniques  that 
were  considered  successful  by  various  studies .  Two  of  the 
four  PA  EAC  methods,  listed  previously,  which  this 
literature  review  focused  on,  were  found  to  be  successful. 
These  were  the  Cumulative  CPI,  and  the  Weighted  Cost  and 
Schedule  methods .  Riedel  and  Chance  used  a  0.8  weight  on 
cost  and  a  0.2  schedule  weighting  in  the  Weighted  Cost  and 
Schedule  EAC  technique  available  in  PA.  This  was  consistent 
with  Air  Force  Material  Command  Pamphlet  (AFMCP)  173-4, 
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which  advocates  the  use  of  the  0.8  and  0.2  weightings  for 
cost  and  schedule,  respectively  (AFMCP  173-4,  1992:19) .  An 
Aeronautical  Systems  Division  (ASD)  reserve  study,  an  Air 
Force  Systems  Command  (AFSC)  EAC  study,  and  an  AD  study  of 
the  AFSC  EAC  showed  this  to  be  a  reliable  forecasting 
formula  (AFMCP  173-4,  1992:19). 

The  identification  of  these  commonly  successful  EAC 
techniques  and  the  accomplishment  of  this  literature  review 
answered  the  first  three  research  objectives : 

1.  Determined  the  EAC  methods  currently  being  used  and 
believed  to  be  successful. 

2 .  Determined  the  work  done  in  the  area  of  estimates  at 
completion,  specifically,  the  index-based  methods . 

3.  Determined  the  scoring  methodologies  that  have  been  used 
to  rate  the  success  of  EAC  forecasting  techniques . 

Even  though  the  techniques  listed  in  this  review  were 
considered  successful,  a  limitation  was  often  highlighted  in 
the  literature:  the  use  of  a  relatively  small  database. 

The  DAES  database  used  in  this  research  tests  the  success  of 
the  techniques  on  a  much  larger  sample  of  contracts . 
Moreover,  a  common  theme  in  these  EAC  techniques  is  the 
dependence  upon  the  Budgeted  Cost  of  Work  Remaining  (BCWR) , 
BAC  -  BCWP .  However,  these  techniques  fail  to  account  for 
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budget  that  may  be  added  to  contracts  because  of  additional 
scope,  cost  overruns,  or  rebaselines.  The  next  chapter  will 
present  the  methodology  used  to  address  this  specific 
problem  as  well  as  accomplish  the  fourth  and  fifth  research 
objectives  of  this  study: 

4.  Use  historical  data  to  develop  BAG  adjustment  factors, 
used  in  conjunction  with  current  EAC  methods,  to  predict 
final  contract  costs  at  various  points  in  the  life  of  DoD 
contracts . 

5 .  Determine  if  the  BAG  adjustment  factors  developed  in 
research  objective  four  improved  the  success  of  the 
prevalent  EAG  methods  determined  in  research  objective  one. 


I1I»  Mfi-tbodoiogy 


Chapter  Overview 

The  methodology  used  to  compare  the  success  of  the 
different  EAC  techniques  in  this  study  involved  simple 
descriptive  statistics.  However,  prior  to  the  actual 
comparison  of  the  different  EAC  techniques,  BAC  adjustment 
factors  were  developed.  These  adjustment  factors  increase 
the  BAC  used  in  EAC  calculations  to  account  for  increases  in 
scope  and  cost  overruns.  A  more  detailed  discussion  of  the 
need  for  these  adjustment  factors  is  included  in  the 
background  section,  later  in  this  chapter.  In  addition,  the 
development  of  these  adjustment  factors  is  described  step  by 
step  in  the  Analysis  Method  section  at  the  end  of  this 
chapter.  The  essence  of  this  chapter,  however,  is  a 
description  of  the  methodology  used  to  compare  the  success 
of  the  various  EAC  techniques,  both  with  and  without  the  BAC 
adjustment  factors . 

Hypothesis 

The  general  hypothesis  of  this  study  was  that  index- 
based  EAC  techniques  provide  better  predictions  of  the 
actual  final  costs  at  completion  of  DoD  contracts  when  used 
in  conjunction  with  the  BAC  adjustment  factors .  In  order  to 
compare  the  success  of  the  different  EAC  techniques,  the 
definition  of  what  makes  an  EAC  method  successful,  when 
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compared  to  other  EAC  methods,  was  developed.  As  can  be 
seen  in  the  literature  review  of  this  research,  prior  EAC 
comparative  studies  used  various  scoring  methodologies  to 
determine  which  EAC  techniques  were  most  successful . 

For  this  study,  success  was  defined  to  include 
accuracy,  bias,  and  what  is  commonly  referred  to  as 
precision.  In  other  words,  not  only  must  the  EAC  technique 
generate  EACs  that  are  close  to  the  costs  at  completion 
(CAC) ,  but  it  must  do  so  consistently  and  without 
unfavorable  bias . 

Testing  Procedures 

Unfortunately,  hypothesis  testing  was  inappropriate  for 
analysis  in  this  study.  This  was  because  of  the 
interdependence  between  each  of  the  EAC  calculations  and  the 
actual  contract  completion  costs .  Each  of  the  index-based 
EAC  techniques  would  be  utilizing  actual  contract  costs . 

This  interdependency  grows  stronger  as  the  contract 
progresses  through  its  life.  For  instance,  at  the  99% 
completion  point,  an  EAC  generated  by  one  of  the  index-based 
techniques  would  consist  almost  totally  of  actual  contract 
costs:  the  ACWP  portion  of  the  EAC  formula  (see  Formulas  2.4 
through  2.7).  This  would  result  in  extremely  high 
collinearity .  These  interdependencies  would  invalidate  any 
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hypothesis  test  used  to  test  the  success  of  an  EAC  at 
predicting  the  contract  completion  cost. 

Various  descriptive  statistics  were  used  to  determine 
the  success  of  an  EAC  method.  Specifically,  an  EAC 
technique  which  produced  a  smaller  mean  absolute  percentage 
error  (MAPE) ,  a  smaller  median  absolute  percentage  error 
(MdAPE) ,  and  which  had  smaller  standard  errors  in  the  MAPE 
or  MdAPE  was  considered  more  successful.  Standard  error  was 
a  good  measure  of  how  consistently  the  technique  was 
successful.  Another  measure  of  accuracy  was  the  number  of 
times  a  technique's  EAC  was  within  10%  of  the  cost  at 
completion . 

Percentage  errors,  as  opposed  to  raw  errors,  were 
chosen  as  decision  criteria  to  account  for  the  varying 
orders  of  magnitude  of  costs  between  contracts .  Absolute 
values  were  selected  to  prevent  positive  and  negative  errors 
from  canceling  each  other  out.  The  reason  for  examining 
the  MdAPE  as  well  as  the  MAPE  was  because  the  MAPE  is 
considered  to  be  too  sensitive  to  outliers  (Armstrong  and 
Collopy,  1992:77).  "The  Median  APE  (MdAPE)  reduces  the  bias 
in  favor  of  low  forecasts,  thus  offering  an  advantage  over 
the  MAPE"  (Armstrong  and  Collopy,  1992:71) . 
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In  addition,  the  EAC  techniques  were  tested  for  bias . 
This  was  accomplished  by  evaluating  the  mean  percentage 
errors  (MPE)  and  the  median  percentage  errors  (MdPE) ;  note 
that  these  are  non-absolute  value  statistics .  Values  other 
than  zero  for  the  MPE  and  MdPE  indicated  bias  in  either  the 
positive  or  negative  direction.  From  a  decision  makers 
standpoint,  perfect  accuracy  in  an  EAC  is  preferred. 

However,  when  planning  for  future  costs,  it  is  evident  that 
a  decision  based  on  slightly  inflated  values  would  be  safer. 
Therefore,  a  technique  with  a  slight  positive  bias  would  be 
favored. 

The  MAPE  for  a  given  EAC  technique  was  defined  to  be 
the  sum  of  the  absolute  percentage  errors  for  all  estimates 
generated  by  an  EAC  technique  divided  by  the  number  of 
estimates  generated  by  that  technique.  As  mentioned 
previously,  the  use  of  a  percentage  error  compensates  for 
the  varying  degrees  of  magnitude  between  contracts .  These 
varying  degrees  refer  to  the  different  dollar  amounts 
utilized  on  each  of  the  contracts.  The  following  is  an 
example  of  how  the  MAPE  was  calculated  for  each  EAC 
technique  in  this  study.  The  EAC  column  represents  EACs 
generated  by  an  EAC  technique,  and  the  Error  column 
represents  the  differences  between  the  generated  EACs  and 
the  contract  costs  at  completion  (CAC) .  The  Absolute  % 

Error  is  defined  as  the  absolute  value  of  the  Error,  divided 
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by  the  CAC,  times  100.  It  is  necessary  to  multiply  the 
Absolute  %  Error  by  100  to  show  it  as  a  percentage,  rather 


than  as  a  decimal.  The  numbers 
illustrative  purposes  only. 


EAC 

CAC 

120 

170 

-50 

1000 

1300 

-300 

550 

750 

-200 

800 

780 

+20 

2500 

3000 

-500 

used  in  this  example  are  for 


Absolute  %  Error 

29.4 
23.1 
26.7 

2.6 

L6.,7 

98.5 


98.5  *  sum  of  Absolute  %  Errors 

98.5  /  5  =»  19.7  =  MAPE 


A  lower  MAPE  for  a  given  EAC  technique,  relative  to  other 
EAC  techniques,  was  deemed  more  successful  (accurate) . 

To  determine  the  median  absolute  percentage  error 
(MdAPE) ,  the  absolute  percentage  errors  (APEs)  were 
calculated  the  Seune  way  as  in  the  example  for  the  MAPE.  The 
APEs  were  then  rank  ordered  from  low  to  high.  The  following 
equations  were  used  to  determine  the  MdAPE: 

MdAPE  =  Observation  —  if  n  is  odd,  or  (3.1a) 

2 

Observation  —  +  1  if  n  is  even,  (3.1b) 

2 

where,  n  »  number  of  estimates  generated. 
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As  mentioned  previously,  the  next  statistic  calculated 
to  compare  the  success  of  the  EAC  techniques  was  the 
standard  errors  for  the  MAPE  and  MdAPE  of  each  EAC 
technique.  This  calculation  was  used  to  incorporate  a 
measure  of  variability  into  the  evaluation  of  the  EAC 
techniques '  success .  The  need  to  assess  variability  was 
derived  from  this  study's  definition  of  E3AC  success, 
wherein,  a  smaller  variation  in  the  MAPE  and  MdAPE  was 
deemed  to  be  more  successful  (the  precision  element  of 
success)  . 


The  computation  of  a  standard  error  for  each  MAPE  and 
MdAPE  was  accomplished  with  the  following  equation: 


g  ^  11^1  (APEt-M(Md)APE)^ 

V  n  -  1 


(3.2) 


where:  s  »  standard  error 

APE  *  Absolute  Percentage  Error 
M(Md)APE  -  Mean  (Median)  Absolute  Percentage  Error 
n  =  number  of  estimates  generated 

A  lower  standard  error  for  a  given  EAC  technique,  relative 

to  other  EAC  techniques,  was  deemed  more  successful 

(precise) . 


The  bias  of  the  EAC  techniques  was  determined  by 
calculating  the  MPE  and  MdPE.  Calculations  for  these 
statistics  were  essentially  the  same  as  for  the  MAPE  and 
MdAPE,  except  the  absolute  value  was  not  talcen  before 
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summing  the  APEs  (see  previous  examples) .  A  negative  value 
for  the  MPE  or  MdPE  indicated  that  the  technique  tended  to 
estimate  lower  than  the  CAC,  while  a  positive  value  would 
indicate  estimates  higher  than  the  CAC.  The  computation  of 
a  standard  error  for  each  MPE  and  MdPE  was  accomplished  with 
the  following  equation: 

,  =  Jg.i  (P&-M(Md)pi7 
II  n  -  1 

where:  s  =■  standard  error 

PE  =■  Percentage  Error 
M(Md)PE  -  Mean  (Median)  Percentage  Error 
n  »  number  of  estimates  generated 

In  this  case,  a  lower  standard  error  indicated  a 
stronger  conclusion  about  the  bias,  meaning,  the  smaller  the 
standard  error,  the  closer  the  values  are  to  the  MPE  and 
MdPE.  Smaller  errors  indicate  a  stronger  bias.  For 
example,  if  the  MPE  or  MdPE  is  a  small  negative  number,  but 
the  standard  error  is  large,  then  it  indicates  that  the 
actual  percentage  errors  range  both  positive  and  negative. 
Whereas,  if  the  MPE  or  MdPE  is  a  small  negative  number  and 
the  standard  error  is  small,  it  is  likely  that  the  majority 
of  the  percentage  errors  are  small  negative  values . 

Data  and  Sample 

The  potential  population  for  this  study  was  all 
completed  DoD  contracts  that  met  Cost  Performance  Report 
(CPR)  reporting  criteria  as  specified  in  DoDI  5000.2.  The 
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sample  data  for  this  study  were  obtained  via  the  Defense 
Acquisition  Executive  Summary  (DAES)  database,  as  collected 
by  the  Office  of  the  Under  Secretary  of  Defense, 

Acquisition,  OUSD (A) .  The  data  consisted  of  cost 
performance  measurement  information  from  536  DoD  contracts 
for  the  period  June  1977  through  June  1993. 

Contract  inclusion  in  this  study  was  based  on  final 
reported  percent  complete  (PC) .  This  study  included  the  534 
DoD  contracts  that  were  at  least  80%  complete.  Percent 
complete  was  determined  by  dividing  the  final  reported 
budgeted  coat  of  work  performed  (BCWP)  by  the  final  budget 
at  completion  (FBAC)  for  the  entire  contract.  This 
information  was  available  in  the  DAES  database.  Note  that 
FBAC  and  BAC  were  identical  amounts  for  the  last  set  of  CPR 
data  reported  for  a  contract.  The  contracts  in  the  database 
varied  with  respect  to  the  final  percent  complete  reported. 
Data  for  some  contracts  was  not  provided  up  to  100% 
completion  point,  while  some  contracts  went  beyond  100% 
complete.  This  was  due  to  thi  fact  that  the  final  reported 
BCWP  was  greater  than  the  final  reported  BAC.  For  purposes 
of  this  study,  contracts  with  percent  completes  computed  to 
be  over  100%,  as  defined  above,  were  assumed  to  be  100% 
complete  and  were  included  in  this  study. 
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The  reason  this  study  included  the  534  DoD  contracts 
that  were  at  least  80%  complete  was  because  contract 
completion  costs  were  needed  to  compare  the  success  of  the 
EAC  techniques .  In  addition,  final  budgets  at  completion 
(FBAC)  were  needed  to  develop  the  BAG  adjustment  factors . 

The  BAG  and  AGWP  for  contracts  in  the  sample  that  were 
reported  as  over  80%  but  under  100%  complete  were 
extrapolated  to  the  100%  completion  point.  Extrapolating 
contract  completion  costs  for  contracts  below  80%  complete 
would  not  contribute  to  the  validity  of  this  research,  as 
the  confidence  in  the  extrapolated  completion  costs  would  be 
low.  Extrapolating  contract  completion  costs  from  GPR  data 
that  did  not  indicate  a  contract  to  be  at  least  80%  complete 
would  have  added  an  extra  level  of  uncertainty  to  the 
analysis . 

For  the  BAG,  this  was  accomplished  by  dividing  the 
total  allocated  budget  (TAB)  by  the  respective  reported 
percent  complete  (PG) .  Similarly,  to  extrapolate  the  AGWP, 
the  actual  cost  of  the  work  performed  (AGWP)  was  divided  by 
the  respective  percent  complete.  For  contracts  computed  to 
be  over  100%  complete,  the  last  government  estimate  at 
completion  (EAG)  was  referenced  as  the  cost  at  completion 
(GAG) .  Similarly,  the  TAB  was  used  as  the  final  budget  at 
completion  (FBAG)  for  contracts  computed  to  be  over  100% 
complete . 
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Contracts  of  all  types,  phases,  and  issuing  service 
were  included  in  the  sample  because  this  study  attempted  to 
develop  BAG  adjustment  factors  that  could  be  applied  to  any 
DoD  contract.  Specifically,  both  fixed  price  and  cost  plus 
contracts  were  included  in  the  sample,  as  well  as  contracts 
from  all  three  services  (Air  Force,  Navy,  and  Army) .  Fixed 
price  contracts  have  fixed  prices  which  the  government 
agrees  to  pay  the  contractor,  regardless  of  the  actual  cost 
of  the  contract.  On  the  other  hand,  in  a  cost  plus 
contract,  the  goveriunent  agrees  to  reimburse  the  contractor 
for  all  allowable  and  reasonable  costs.  In  addition,  both 
research  and  development  (R&O) ,  as  well  as,  production 
contracts  were  included.  In  a  research  and  development 
contract,  the  government  usually  pays  for  concepts  or  paper 
designs.  In  other  words,  the  contractor  spends  a  majority, 
if  not  all,  of  the  contract  funds  on  designing  and 
developing  a  weapon  system.  Whereas,  in  a  production 
contract,  the  majority  of  the  contractual  effort  is  spent  on 
manufacturing  and  assembling  the  weapon  system. 

Furthermore,  the  data  were  broken  into  quart ile  percent 
completion  points  to  compare  the  success  of  the  different 
EAC  techniques  at  different  contract  completion  stages . 
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Pata  Limitationa 


The  data  used  in  this  analysis  were  initially  generated 
by  Defense  Department  contractors.  As  such,  data 
reliability  and  validity  were  dependent  upon  the  DoD 
contractors'  correct  utilization  of  their  Performance 
Measurement  Systems  (PMS) .  As  discussed  in  Chapter  Two,  it 
was  assumed  that  contractors '  PMS  systems  were  validated  by 
an  official  government  review  team.  Therefore,  it  was 
assumed  that  the  data  originally  produced  by  the  Defense 
contractors  were  reliable. 


Background  for  the  BAG  Adjustment  Factors 

The  development  of  the  BAG  adjustment  factors  stems 
from  the  basic  method  in  which  index-based  EAC  techniques 
estimate  the  final  cost  of  contracts.  The  following  is  a 
somewhat  generic  formula  which,  in  one  form  or  another,  most 
index-based  EAC  techniques  follow: 


EAC 


ACWP  + 


BAG  -  BCWP 
INDEX 


(3.5) 


The  standard,  or  shared,  variables  among  most  index-based 
EAC  methods  in  this  formula  are  the  ACWP  and  the  BCWP .  By 
definition,  ACWP  and  BCWP  are  known  dollar  amounts  since 
they  describe  the  results  of  costs  that  have  already  been 
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incurred  or  work  that  has  already  been  performed.  On  the 
other  extreme,  it  is  the  Index  in  the  generic  formula  (3.5) 
that  differentiates  one  index-*based  method  from  another. 

Some  of  the  various  Indexes  incorporate  past  cost 
performance,  past  schedule  performance,  or  combinations  of 
both,  in  an  attempt  to  predict  the  final  cost  at  completion 
of  a  contract.  The  development  of  an  Index  can  be  simple  or 
extremely  complicated,  such  as  in  the  NAVSEA  90 '  s  Foimiula 
(2.6).  The  EAC  comparative  studies  examined  in  this  study's 
literature  review  attempted  to  discover  which  indexes  were 
reported  to  most  successfully  estimate  final  contract 
completion  costs .  Unfortunately,  the  Index  is  not  the  only 
variable  in  the  generic  index-based  EAC  formula  (3.5)  that 
is  not  fixed,  or  known  with  certainty. 

At  the  time  an  estimate  is  made,  a  contract's  final  BAC 
is  almost  never  known.  Historically,  BAC  grows  through  the 
life  of  a  contract  due  to  added  scope  from  changing 
requirements .  This  fact  makes  the  use  of  BAC  in  the  generic 
formula  (3.5)  disputable.  "Rebaselining  a  contract  may  be 
necessary  for  cost  control  when  budgets  become  unrealistic 
at  the  work  package  level,  but  it  causes  problems  when  using 
C/SCSC  reports  to  measure  and  predict  performance"  (Bright 
and  Howard,  1981:10) .  Therefore,  the  use  of  BAC,  as  it  is 
known  at  the  time  of  an  estimate  at  completion  is  made,  will 
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almost  undoubtedly  result  in  an  unrealistically  low  estimate 
of  the  final  contract  completion  cost.  This  is  due  to  the 
fact  that  the  amount  of  work  remaining,  the  budgeted  cost  of 
work  remaining,  or  BCWR  (BAG  -  BCWP),  will  be  understated  if 
the  BAG  is  understated. 

In  order  to  account  for  not  knowing  what  the  final  BAG 
will  be,  a  certain  number  of  replacements  for  BAG  in  the 
generic  formula  (3.5)  have  been  recommended.  One 
replacement  for  BAG  is  the  Latest  Revised  Estimate  (LRE) . 
This  approach  can  account  for  future  budget  which  may  be 
approved  on  the  contract,  but  which  has  not  been 
incorporated  into  the  contractor's  performance  measurement 
system.  In  addition,  using  the  LRE  can  account  for  past 
cost  overruns  as  well  as  future  cost  overruns  (future  cost 
overruns  only  to  the  extent  a  contractor  is  willing  to 
repoart  them  in  his  LRE)  .  Other  EAG  techniques,  like  the 
NAVSEA  90 's  formula  (2.6),  attempt  to  account  for  the  use  of 
Management  Reserve  (MR)  and  Over  Target  Baselines  (0TB) . 
However,  these  replacements  for  BAG  still  fail  to  account 
for  unknown  budget  which  could  be  added  to  the  contract  in 
the  future.  For  this  reason,  the  BAG  adjustment  factors 
developed  as  part  of  this  research,  attempt  to  account  for 
these  unknown  budget  additions,  based  on  historical  data. 
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If  the  BAG  adjustment  factors  improve  the  success  of 
EAC  forecasting  techniques,  then  their  benefits  will  be  two¬ 
fold.  First,  it  provides  the  program  manager,  as  well  as 
higher  level  management,  a  potentially  more  accurate 
projection  of  the  budget  or  scope  that  might  be  added  to  the 
contract  before  completion.  Second,  they  can  help  provide 
the  program  manager,  and  higher  level  management,  a 
potentially  accurate  forecast  of  the  cost  at  completion  for 
the  contract. 

Analysis  Method 

The  exact  methodology  used  in  this  study  is  presented 
in  a  step  by  step  fashion  to  facilitate  its  understanding. 

1 .  The  DAES  database  was  obtained  from  OUSD (A) ,  and  was 
generally  reviewed  for  content  and  suitability.  This 
included  reviewing  data  for  completeness,  especially  the 
parameters  necessary  to  utilize  the  various  techniques . 

2 .  Data  were  sampled  based  on  the  criteria  outlined  in  the 
Data  and  Sample  section  of  this  chapter.  This  lead  to  the 
exclusion  of  two  contracts  that  were  not  reported  as  at 
least  80%  complete. 


3.  The  BACs  and  ACWPs  for  the  contracts  that  were  reported 
over  80%  complete,  but  less  than  100%  complete,  were 
extrapolated  to  the  100%  completion  point . 

4 .  The  C/SCSC  contract  data  were  separated  into  quartiles . 
They  were  further  separated  by  contract  type,  meaning  fixed 
price  or  cost  plus .  This  categorization  was  selected 
because  the  situation  surrounding  the  two  types  of 
contracts  allow  or  restrict  costs  in  different  ways .  For 
instance,  a  fixed  price  contract  tends  to  restrict  costs  to 
an  initially  agreed  upon  amount,  while  cost  plus  contracts 
reimburse  the  contractor  for  all  allowable  costs  incurred. 
The  two  contract  philosophies  seem  quite  different, 
therefore  the  respective  adjustment  factors  may  be  different 
as  well.  The  analysis  was  accomplished  by  quartile  for  the 
entire  sample,  as  well  as  for  fixed  price  and  cost  plus  type 
contracts  independently.  Each  of  the  following  outlined 
procedures  was  accomplished  for  each  quartile  for  the  entire 
sample,  as  well  as  each  category. 

5 .  The  factors  to  account  for  additional  budget  being  added 
to  a  contract  (which  are  used  in  conjunction  with  different 
index-based  EAC  techniques  to  estimate  contract  completion 
costs)  were  developed. 

a.  A  ratio  of  the  difference  (Difference  Ratios) 

between  the  final  budget  at  completion  and  the  BAC 
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at  the  time  the  C/SCSC  data  were  reported  was 
computed:  (FBAC  -  BAG) /BAG. 

b.  The  Difference  Ratios  computed  in  part  5a  were 
sTommed. 

c.  The  mean  of  the  Difference  Ratios  was 


calculated. 


d.  The  Mean  Difference  Ratios  calculated  in  part 


5c  were  changed  into  factors  by  adding  one  to 
them.  The  BAG  of  a  contract  can  be  multiplied  by 


these  factors  (the  BAG  adjustment  factors), 
thereby  producing  an  estimate  of  what  the  final 
BAG  is  going  to  be  (EFBAG) . 


The  following  is  an  example  of  how  these  factors  were 


computed.  Difference  Ratio  is 


BAG (0-25%  Quart ile) 

50  100 
500  800 
350  450 
125  120 
250  600 


defined  as  (FBAG  -  BAG) /BAG. 

Piffsrense .Ratio 

1.000 

0.600 

0.286 

-0.040 

L>4.0.Q 

3.246 


3.246  »  sum  of  Difference  Ratios 

3.246  /  5  »  0.649  »  Mean  Difference  Ratio 
1.0  +  0.649  *  1.649  »  BAG  Adjustment  Factor 
In  this  example,  the  mean  difference  ratio  shows  that  the 
average  BAG  is  nearly  65%  lower  than  the  FBAC.  Therefore, 
the  BAG  should  be  increased  by  a  factor  of  1.649. 
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6.  The  selected  EAC  techniques:  cumulative  CPI,  weighted 
cost  and  schedule,  NAVSEA  90,  and  the  schedule  cost  index 
(SCI) ,  were  utilized  with  the  data  set  to  project  estimates 
at  complete  (EACs) .  The  formulas,  numbered  2.4,  2.5,  2.6, 
and  2.7,  respectively,  are  listed  in  chapter  two  of  this 
study . 

7 .  The  selected  EAC  techniques  were  used  in  conjunction 
with  the  BAC  adjustment  factors  created  in  step  5  above  to 
project  EACs . 

8.  A  comparative  analysis  was  accomplished.  The  Mean 
Absolute  Percentage  Error  (MAPE)  and  Median  Absolute 
Percentage  Error  (MdAPE) ,  as  well  as  their  standard  errors, 
were  calculated  for:  1)  the  EAC  methods  recommended 
through  review  of  applicable  literature,  and  2)  the  same  EAC 
techniques  adjusted  using  the  factors  developed  in  step  5 . 
The  methods  were  compared  to  determine  which  method  was  more 
successful  (i.e.  lowest  MAPE,  lowest  MdAPE  and  lowest 
standard  error) . 

9.  The  number  of  times  an  EAC  was  within  10%  of  the  CAC  was 
tabulated.  Conclusions  were  drawn  as  to  which  techniques 
were  more  consistently  within  this  range. 

10.  A  comparative  analysis  was  accomplished.  The  Mean 
Percentage  Error  (MPE)  and  Median  Percentage  Error  (MdPE) , 
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as  well  as  their  standard  errors,  were  calculated  for: 

1)  the  EAC  methods  recommended  through  review  of  applicable 
literature,  and  2)  the  same  EAC  techniques  adjusted  using 
the  factors  developed  in  step  5.  The  methods  were  compared 
to  determine  which  methods  were  biased  (i.e.  higher  MPE  and 
higher  MdPE) .  The  lower  the  standard  error,  the  stronger 
the  bias  conclusion.  Conclusions  were  drawn  about  which 
methods  were  more  biased.  For  example,  if  MPE  or  MdPE  was 
-5.5  and  had  a  standard  error  of  3.0,  the  technique  would  be 
deemed  bias  in  the  negative  direction.  However,  if  the 
standard  error  was  45,  then  it  would  indicate  that  the 
percentage  errors  were  both  positive  and  negative, 
decreasing  the  ceirtainty  of  the  bias  conclusion. 

11.  Recommended  EAC  methods  that  proved  to  be  successful  as 
defined  in  this  study  (i.e.  more  accurate  and  less 
unfavorably  biased)  for  the  various  quartiles,  for  the 
entire  contract  life,  and  for  the  various  contract 
categories . 
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Chapter  III  outlined  the  methodology  followed  in 
testing  the  success  of  the  EAC  techniques.  This  chapter  is 
a  presentation  and  discussion  of  the  results  from 
accomplishing  the  prescribed  methodology.  This  chapter  is 
sectioned  by  categories  of  contracts  evaluated:  non- 
categorized,  cost-type  contracts,  and  fixed  price  contracts. 
They  are  further  categorized  by  completion  stage  quartile. 
The  results  for  each  of  the  categorizations  are  reported  in 
a  similar  manner.  For  each  categorization,  a  table  is 
provided  with  the  specific  niamerical  results  of  the 
evaluation.  Accompanying  each  table  are  graphs  depicting 
the  relationship  of  the  descriptive  statistics  (MAPE,  MdAPE, 
etc.)  for  each  of  the  EAC  techniques.  Preceding  all  of  the 
tables  and  graphs  is  a  brief  discussion  of  the  results  of 
this  analysis. 

The  first  of  the  three  graphs  accompanying  each  table 

includes  the  measures  of  accuracy  and  precision:  Mean 

Absolute  Percent  Error  <MAPE) ,  Median  Absolute  Percent  Error 

(MdAPE) ,  and  their  respective  standard  errors  (SE) .  The 

second  graph  for  each  category  includes  the  bias  measures : 

Mean  Percent  Error  (MPE) ,  Median  Percent  Error  (MdPE) ,  and 

their  respective  standard  errors  (SE) .  The  final  graph 

shows  the  results  of  counting  how  many  times  an  EAC 
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technique  predicted  a  cost  that  was  within  10%  of  the  actual 
final  contract  completion  cost. 

BAG.  Adiuatment  JFactor  (BAF^  Development 

Before  discussing  the  results  of  the  EAC  comparison, 
the  development  of  the  BAG  adjustment  factors  (BAFs)  is 
presented.  Table  4-1  shows  the  information  that  was 
generated  to  calculate  the  BAG  adjustment  factors  for  each 
quartile:  the  sum  of  the  errors  (between  the  EAG  and  the 
GAG)  and  the  niimber  of  GPR  line  items  in  the  database  for 
which  an  EAG  was  computed.  Please  refer  to  step  5  of  the 
Analysis  Method  Section  of  Ghapter  3  for  a  more  detailed 
discussion  of  how  these  factors  were  developed. 

As  indicated  in  Table  4-1,  separate  BAG  adjustment 
factors  were  developed  for  each  quartile  of  the  non- 
categorized,  cost-type,  and  fixed  price  contracts.  As 
expected,  the  BAG  adjustment  factors  decreased  going  from 
the  first  quartile  to  the  last  quartile.  This  was  expected 
because  contracts  that  are  still  early  in  their  life-cycle 
are  more  likely  to  experience  budget  additions  or  cost 
overruns  before  their  completion,  than  are  contracts  late  in 
their  life-cycle.  Intuitively,  a  contract  that  is  only  10% 
complete  will  experience  more  contractual  and  technical 
requirements  changes  before  its  completion,  than  a  contract 
that  is  already  90%  complete.  Noteworthy  also,  the  BAFs  are 
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somewhat  higher  for  the  cost-type  contracts  than  they  are 
for  the  fixed  price  contracts . 


Table  4-1 

BAG  ADJUSTMENT  FACTORS 


All  Contracts  (Non-cateqorized) 


%  Complete 

0-25% 

26-50% 

Sum  Errors 

537.9828 

500.4079 

544.2819 

837.5972 

Line  Items 

1001 

1197 

1739 

3234 

1.5374 

1.4181 

1.3130 

1.2590 

Cost-Type  Contracts 

%  Complete 

0-25% 

26-50% 

51-75% 

76-100% 

Svun  Errors 

202.3507 

226.2420 

231.7277 

362.0979 

Line  Items 

278 

382 

633 

1312 

Factor 

1.7279 

1.5923 

1.3661 

1.2760 

Fixed 

Price-Type  Contracts 

%  Complete 

0-25% 

26-50% 

51-75% 

76-100% 

Sum  Errors 

335.6321 

274.0152 

311.7049 

474.4430 

Line  Items 

723 

814 

1101 

1915 

Factor 

■gltLi.l1 

1.3366 

1.2831 

1.2478 

Notes : 

Sum  Errors  -  the  sum  of  the  difference  between  the  FBAC  and 
BAG  at  the  time  the  data  was  reported  for  all  line  items . 

Line  Items  -  total  number  of  data  points  (CPRs  reported)  for 
that  quartile. 

Factor  -  (Sum  Errors  /  Line  Items)  +  1 


The  next  section  of  this  chapter  presents  the  results 
of  the  EIAC  comparative  analysis.  As  previously  mentioned, 
the  tables  and  graphs  are  sectioned  by  category  of  contract 
evaluated:  non-categorized,  cost-type  contracts,  and  fixed 
price  contracts.  The  sections  are  further  categorized  by 
completion  stage  quartile. 
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Results  and  Discussion 


As  the  accuracy /precision  figures  depict,  the 
techniques  without  the  BAFs  were  more  accurate  and  precise. 
These  figures  also  reveal  that  the  NAVSEA  90  formula  had  a 
slightly  lower  MAPE  and  MdAPE,  and  was,  therefore,  slightly 
more  accurate  and  precise  than  the  other  methods  for  each 
contract  category  and  quartile.  The  SCI  formula  was 
typically  the  second  most  successful  in  terms  of  accuracy 
and  precision. 

The  bias  figures  represent  the  relative  bias  of  the 
different  EAC  techniques.  These  are  the  second  figures 
associated  with  each  table.  The  four  techniques  without  the 
BAFs  were  always  negatively  biased,  while  the  addition  of 
the  BAFs  always  caused  a  positive  bias.  Furthermore,  while 
at  the  aggregate  level,  the  addition  of  the  BAFs  caused  a 
positive  bias  (positive  MPE  and  MdPE) ,  the  large  SEs 
indicated  that  the  actual  range  most  likely  included  both 
negative  and  positive  errors.  Similarly,  the  EAC  techniques 
without  the  BAFs  also  had  large  SEs  relative  to  their  MPEs 
and  MdPEs  indicating  that  the  range  of  errors  was  both 
positive  and  negative. 

An  additional  measure  of  accuracy  and  precision  was  the 

number  of  times,  converted  to  a  percent,  the  Absolute 

Percent  Error  (APE)  between  the  EAC  and  CAC  was  less  than  or 

equal  to  10%.  This  measure  is  depicted  in  the  figures 
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entitled,  APE  <  10%.  These  are  the  third  figures  associated 
with  each  table.  Using  this  measure,  the  NAVSEA  90  formula 
EACs  fell  within  10%  of  the  CACs  more  often  than  the  other 
techniques.  The  SCI  technique  was  again,  most  often,  a 
close  second.  Additionally,  inclusion  of  the  BAFs,  decreased 
the  accuracy  of  the  EAC  techniques. 

In  general,  the  results  of  this  study  indicate  that  the 
NAVSEA  90  technique  was  the  most  successful.  The  results 
were  unaffected  by  contract  category:  noncategorized,  cost- 
type,  and  fixed  price.  In  addition,  the  results  were  the 
same,  regardless  of  the  contract  quartile  examined. 


First  Quartlle  -  Noncategorlzed  (1001  Line  Items) 
BAG  Adjustment  Factor  (BAF)  »  1.5374 


6 


N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


Table  4-3 

Second  Quartile  -  Noncategorized  (1197  Line  Items) 
BAG  Adjustment  Factor  (BAF)  -  1.4181 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 
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Figure  4-6 
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Third  Quartile  -  Noncategorlzed  (1739  Line  Items) 
BAG  Adjustment  Factor  (BAF)  ■■  1.3130 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 
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Figure  4-7 
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Figure  4-8 


«oxm 

soom 


•  4o.an 


e  30.00% 


n  20n0% 


t 


10.00% 


0.00% 


APE  <  10%  for  (ho  THIRD  OUARTILE  -  NONCAT EGORIZ ED 

lUL... 


wc/s 


N90 


SO  Cri«CAr  WC/S«tAF  N90*tAf  Sa+BAf 


Table  4-5 

Fourth  Quartile  -  Noncategorlzed  (3234  Line  Items) 
BAG  Adjustment  Factor  (BAF)  -  1.2590 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 
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Figure  4-12 
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Table  4-6 

First  Quartlle  -  Cost  Contracts  (278  Line  Items) 
BAC  Adjustment  Factor  (BAF)  1.7279 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


Table  4-7 

Second  Quartlle  -  Cost  Contracts  (382  Line  Items) 
BAC  Adjustment  Factor  (BAF)  «  1.5923 
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SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 
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Figure  4-16 
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Table  4-8 

Third  Quartile  -  Cost  Contracts  (633  Line  Items) 
BAC  Adjustment  Factor  (BAF)  »  1.3661 
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SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


ACCURACY  /PREOSIONforthcTHIRO  OUARTILE  -COST  CONTRACTS 
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Figure  4-  19 
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Table  4-9 

Fourth  Quartile  -  Coat  Contracts  (1312  Line  Items) 
BAC  Adjustment  Factor  (BAF)  -  1.2760 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 
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Figure  4-24 

4-21 


Table  4-10 

First  Quartile  -  Fixed  Price  Contracts  (723  Line  Items) 
BAG  Adjustment  Factor  (BAF)  1.4642 
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N90+DAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


Figure  4-26 
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Figure  4-27 
4-23 


Table  4-11 

Second  Quartile  -  Fixed  Price  Contracts  (814  Line  Items) 
BAC  Adjustment  Factor  (BAF)  -  1.3366 
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N90+BAF  -  NAVSEA  90  +  (BAF) 

SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


ACCURACY  /PREaSION forth* SECOND  OUARTILE  -  FIXED  PRICE 

CONTRACTS 


Figure  4-28 


Figure  4-29 
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Figure  4-30 
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Table  4-12 

Third  Quartile  -  Fixed  Price  Contracts  (1101  Line  Items) 
BAG  Adjustment  Factor  (BAF)  -  1.2831 
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SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


ACCURACY  /PREOSION forth* THIRD  QUARTILE  -  FIXED  PRICE 
CONTRACTS 


APE  <  10%  forth*THIRD  QUARTILE  -FIXED  PRICE  CONTRACTS 


Figure  4-33 
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Table  4-13 

Fourth  Quartile  -  Fixed  Price  Contracts  (1915  Line  Items) 
BAG  Adjustment  Factor  (BAF)  »  1.2478 
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SCI+BAF  -  Schedule  Cost  Index  (SCI)  +  (BAF) 


ACCURACY  /PREaStON  for  the  FOURTH  QUARTILE  -FIXED  PRICE 

CONTRACTS 
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Figure  4-36 


Chapter  Summary 


This  chapter  provided  the  results  of  the  analysis 
performed  in  this  research  effort.  Commonly  used,  index- 
based  EAC  techniques  were  compared,  both  with  and  without 
newly  developed  BAC  adjustment  factors  (BAFs) .  The 
comparison  included  the  calculation  of  the  following 
descriptive  statistics:  MAPE,  MdAPE,  MPE,  and  MdPE,  along 
with  their  respective  standard  errors  (SEs) -  In  addition, 
the  number  of  times  an  EAC  technique  prediction  came  within 
10%  of  the  CAC  was  also  counted. 

The  statistics  were  used  to  compare  the  success  of  the 
different  EAC  techniques.  In  Chapter  Three,  success  was 
defined  to  include  accuracy,  bias,  and  precision.  The 
following  chapter.  Chapter  Five,  discusses  the  conclusions 
that  were  drawn,  and  recommendations  that  were  made,  based 
upon  the  analysis  performed  in  this  chapter.  Chapter  Four. 
Chapter  Five  will  also  list  some  recommendations  for  further 
research  in  this  area. 
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V.  Conclusions  and  RfeCQtnmenda.tJ.Qna 


Chapter  Overview 

Stunmary  findings  and  conclusions  of  this  research 
effort  are  presented  in  this  chapter.  First,  a  brief 
restatement  of  the  findings  from  Chapters  One,  Tv/o,  and 
Three,  and  a  summary  of  the  results  from  Chapter  Four  are 
provided.  Next,  general  conclusions  regarding  this  overall 
research  effort  are  discussed.  This  discussion  includes  a 
comparison  of  the  results  of  this  research  effort  to  the 
conclusions  of  the  literature  reviewed  in  Chapter  Two. 
Finally,  reconuaendations  on  areas  for  future  related 
research  are  presented. 

Review  of  Chapters  One.  Two.  Three,  and  Four 

Chapter  One.  This  chapter  introduced  the  problem  of 
contract  cost  overruns  in  the  Defense  Industry.  In 
addition,  it  pointed  out  that  better  estimates  at  completion 
for  DoD  contracts  can  help  top  defense  and  other  national 
leaders  make  better  or  more  educated  decisions  in 
restructuring  the  U.S.  defense  budget.  Chapter  One  guided 
this  research  effort  towards  developing  a  better  technique 
to  estimate  contract  completion  costs  for  DoD  contracts . 

Chapter  Two.  This  chapter  summarized  the  findings  of 
the  literature  review  accomplished  for  this  research  effort. 
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The  primary  purposes  of  reviewing  other  literature  were  to 
avoid  duplicating  research  already  accomplished  and  to 
discover  what  areas  of  EAC  research  needed  further 
attention.  The  literature  review  revealed  that  no  single 
EAC  method  outperformed  all  other  techniques  all  of  the 
time.  Different  studies  recommended  that  different  EAC 
techniques  were  appropriate  at  different  times  in  a 
program's  life-cycle.  In  addition,  the  literature  review 
identified  the  various  scoring  methodologies  that  have  been 
used  to  compare  the  success  of  EAC  techniques .  The 
literature  review  helped  to  answer  the  first  three  research 
objectives : 

1 .  Determined  the  EAC  methods  currently  being  used  and 
believed  to  be  successful. 

2 .  Determined  the  work  done  in  the  area  of  estimates  at 
completion,  specifically,  the  index-based  methods. 

3 .  Determined  the  scoring  methodologies  that  have  been  used 
to  rate  the  success  of  EAC  forecasting  techniques . 

Chapter  Three.  This  chapter  outlined  the  methodology 
used  to  compare  the  success  of  the  different  EAC  techniques 
included  in  this  research  effort.  It  also  identified  the 
methodology  used  to  develop  the  BAC  adjustment  factors .  The 
BAC  adjustment  factors  were  used  in  EAC  calculations  to 
account  for  increases  in  cost  overruns  and  scope  additions . 
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In  general,  accuracy /precision  was  compared  by  computing  the 
Mean  Absolute  Percentage  Error  (MAPE) ,  Median  Absolute 
Percentage  Error  (MdAPE) ,  and  their  respective  standard 
errors .  Bias  was  compared  through  the  use  of  the  Mean 
Percentage  Error  (MPE) ,  Median  Percentage  Error  (MdPE) ,  and 
their  respective  standard  errors  -  As  an  additional  measure 
of  accuracy,  the  number  of  times  a  technique  estimate  was 
within  10%  of  the  actual  final  contract  completion  cost  was 
tabulated. 

Chapter  Four.  This  chapter  provided  the  results  of  the 
analysis  performed  in  this  research  effort.  Commonly  used, 
index-based  EAC  techniques  were  compared,  both  with  and 
without  newly  developed  BAC  adjustment  factors  (BAFs) .  The 
comparison  included  preparation  of  tables  and  figures 
depicting  the  following  descriptive  statistics;  MAPE, 

MdAPE,  MPE,  and  MdPE  along  with  their  respective  standard 
errors  (SEs) .  The  number  of  times  an  EAC  technique  came 
within  10%  of  the  CAC  was  also  counted  and  graphically 
depicted.  In  addition,  research  objective  four  was 
accomplished  within  this  chapter; 

4.  Using  historical  data,  BAC  adjustment  factors  were 
developed  which  were  used  in  conjunction  with  current  EAC 
methods  to  predict  final  contract  costs  at  various  points  in 
the  life  of  DoD  contracts . 
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General  Conclusions 


The  overriding  conclusion  of  this  research  effort  was 
that  the  BAG  adjustment  factors  did  not  improve  the  success 
of  the  existing  index-based  EAC  techniques.  This  conclusion 
is  overwhelmingly  obvious  after  reviewing  the  tables  and 
figures  in  Chapter  Four.  This  general  conclusion  answered 
the  fifth  and  final  research  objective: 

5 .  Determined  if  the  BAG  adjustment  factors  developed  in 
research  objective  four  improved  the  success  of  the 
prevalent  EAC  methods  determined  in  research  objective  one. 

Beyond  the  primary  focus  of  this  research  effort,  which 
was  to  test  whether  or  not  the  BAFs  improved  an  EAC 
technique's  success,  it  was  possible  to  extract  additional 
general  conclusions.  The  following  discussion  focuses  on 
the  ELAC  techniques  without  the  inclusion  of  the  BAFs . 

NAVSEA  90 .  The  NAVSEA  90  technique  was  clearly  the 
most  successful  technique  examined.  It  performed  as  good  or 
better  than  all  the  other  techniques  in  all  contract 
quartiles  and  contract  categories;  noncategorized,  cost- 
type,  and  fixed  price.  Unfortunately,  none  of  the  previous 
EAC  studies  reviewed  in  Chapter  Two  included  the  NAVSEA  90 
technique . 
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Schedule  Cost  Index  rsci) .  This  technique  was  also 
successful.  It  came  in  a  close  second  to  the  NAVSEA  90 
technique  in  many  of  the  contract  quartiles  and 
categorizations.  Although  the  SCI  technique  never  really 
outperformed  the  NAVSEA  90  formula,  its  performance  did 
improve  in  the  latter  contract  quartiles .  This  conclusion 
reaffirms  the  conclusion  of  two  previous  EAC  studies :  the 
Covach  study  and  the  Bright  and  Howard  research.  Both  of 
these  studies  recommended  the  SCI  technique. 

Weighted  Cost  and  Schedule.  This  EAC  technique  was  not 
as  successful  as  the  NAVSEA  90  or  SCI  techniques.  However, 
on  occasion,  the  Weighted  Cost  and  Schedule  technique 
outperformed  the  SCI  technique,  but  was  never  better  than 
the  NAVSEA  90  technique.  The  Bright  and  Howard  research 
concluded  that  the  Weighted  Cost  and  Schedule  technique  was 
successful  in  the  early  contract  phases.  The  Price  study 
recommended  the  Weighted  Cost  and  Schedule  technique 
throughout  the  life  of  the  contract.  The  Bright  and  Howard 
study  conclusions  and  the  Price  study  conclusions  were 
contrary  to  the  findings  in  this  study.  This  is  most  likely 
due  to  the  fact  that  neither  study  included  the  NAVSEA  90 
formula.  Furthermore,  the  Price  study  didn't  even  include 
the  SCI  formula. 
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Cum  CPI 


was  the  least  successful  technique  examined  in  this  research 
effort .  The  large  negative  bias  suggests  that  this  EAC 
technique  almost  always  produced  low  estimates .  This  means 
that  a  contract  will  more  than  likely  have  an  actual  cost  at 
completion  (CAC)  that  is  higher  than  an  EAC  generated  with 
the  Cum  CPI  technique.  Although  this  conclusion  might  be 
drawn  about  the  other  three  techniques,  the  results  of  this 
research  suggest  that  it  is  more  clearly  the  case  for  the 
Cum  CPI  technique.  Nearly  all  of  the  previous  EAC  studies 
stated  that  the  Cum  CPI  was  a  good  technique  either  overall 
or  in  specified  contract  life-cycle  phases  (see  Table  2-1) . 
However,  this  is  probably  due  to  the  fact  that  none  of  them 
included  the  NAVSEA  90  technique,  and  a  few  of  them  did  not 
include  the  SCI  technique. 

It  should  be  noted  that  the  results  of  this  research 
may  differ  from  previous  studies  because  of  the  different 
databases  that  were  used.  In  addition,  the  scoring 
methodology  to  determine  which  EIAC  techniques  were 
successful  was  different  than  the  methodologies  used  in 
previous  EAC  research  efforts. 

The  generic  index  based  formula  (3.5),  discussed  in 
Chapter  Three,  revealed  that  an  EAC  is  dependent  upon  how 
much  work  remains  to  be  done  on  a  contract .  On  DoD  weapons 
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system  contracts,  this  is  know  as  the  Budgeted  Cost  of  Work 
Remaining  (BAG  -  BCWP  =  BCWR) .  Although  the  addition  of  the 
BAG  adjustment  factors  (BAFs)  diminished  the  success  of  the 
EAC  techniques,  the  concept  of  adjusting  the  remaining 
budget  on  a  contract  (BCWR)  for  reasons  of  added  scope  or 
cost  overruns  was  not  entirely  unsupported. 

It  was  concluded  in  this  study  that  the  success  of  the 
EAC  techniques,  when  used  in  conjunction  with  the  BAFs,  was 
drastically  degraded  because  of  the  magnitude  of  the 
positive  bias  resulting  from  the  BAFs .  Too  much  positive 
bias  skews  the  estimate  to  the  high  side.  The  inflated 
estimate  resulting  from  the  BAF  is  probably  more  unrealistic 
than  the  low  estimates  from  the  unadjusted  techniques . 

Areas  for  Future  Related  Research 

In  general,  the  concept  of  adjusting  the  remaining 
budget  on  a  contract  (BCWR)  for  reasons  of  added  scope  or 
cost  overruns  when  using  an  index-based  EAC  technique  needs 
further  exploration.  The  contractor's  performance  to  date 
is  reflected  in  the  Index  that  is  chosen  in  the  generic 
index-based  formula  (3.5).  Having  the  capability  to  include 
an  estimate  of  the  remaining  work,  as  well  as  the 
contractor's  performance  to  date,  would  be  extremely 
beneficial.  Specifically,  a  different  methodology  to 
develop  the  BAFs  is  in  order.  Given  the  amount  of  positive 
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bias  that  resulted  from  the  methodology  used  in  this 
research,  an  examination,  and  possible  deletion,  of  extreme 
outliers  in  the  database  is  another  area  that  might  be 
explored . 

In  addition,  in  this  study,  there  were  only  four 
distinct  BAFs  corresponding  with  contract  quartiles:  0-25%, 
26-50%,  51-75%,  and  76-100%  complete.  This  was  done  for 
each  contract  category:  noncategorized,  cost-type,  and 
fixed  price  (see  Table  4-1)  .  Another  methodology  might  be 
to  construct  an  equation  for  the  BAFs,  with  percent  complete 
as  the  independent  variable,  and  the  BAF  as  the  dependent 
variable.  This  would  help  smooth  out  the  application  of  the 
BAFs.  The  following  is  a  generic  formula  representing  this 
relationship: 

BAF  =  f (Percent  Complete)  (5.1) 

Future  research  might  focus  on  a  different  equation  for  each 
contract  category:  noncategorized,  cost-type,  and  fixed 
price. 

Finally,  additional  contract  categories  might  be 
established.  For  example,  BAFs  could  be  developed  and 
analyzed  for  Air  Force,  Army,  and  Navy  contracts. 
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Additionally,  Research  and  Development  (R&D)  and  Production 
contracts  could  be  separated. 


■t 
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